Don't convert param buffer to utf-8 · Pull Request #165 · hgourvest/node-firebird

ghost · 2018-10-28T15:23:34Z

When a buffer is used as parameter it would always be converted to a
utf-8 string. To allow working with a DB that has charset NONE, the
buffer is now send directly without converting to string.
This could solve #164

You have to take care of your own encoding.

It could be used like this:

const Firebird = require('node-firebird')
const windows1252 = require('windows-1252');

function encodeString(s) {
    return Buffer.from(windows1252.encode(s), 'binary')
}

function decodeString(b) {
    return windows1252.decode(b.toString('binary'))
}

Firebird.attach(options, (err, db) => {
    if (err) throw err;
 
    db.query('INSERT INTO TABLE (NR, FOO) VALUES(?, ?)', 
             [1, encodeString('19 €')], (err, result) => {
        db.query('SELECT * FROM TABLE WHERE NR=?', [1], (err, result) => {
            console.log(decodeString(result[0].FOO))	
            db.detach();
        });
    });
});

When a buffer is used as parameter it would always be converted to a utf-8 string. To allow working with a DB that has charset NONE, the buffer is now send directly without converting to string.

e-baron · 2019-01-07T14:31:08Z

Hello,
Thank you very much for this update !!! I have quickly tested it and it worked well. I am very happy as I won't need to use ODBC to work with a DB with NONE charset.

joaodforce · 2020-03-31T18:57:54Z

I wanted to merge this into my Fork, but the OP deleted his Repo fork :/

ghost · 2020-03-31T19:17:25Z

Sorry about that. I cleaned up too much (PR is a year old). But I think you can still merge a PR without the fork?

joaodforce · 2020-03-31T19:20:27Z

The Owner can merge this PR on his master, but I tried to create a PR on my Fork but I could not find your branch do cherrypick a Commit.

I tried adding the original REPO as a second remote to my fork, and fetched it, but your commit doesn't appear. so It seems the PR's commit exists outside the main Repo.

I'm trying to merge the changes by hand lol.

ghost · 2020-05-02T17:59:31Z

I can make a new PR for this if you like?

joaodforce · 2020-05-02T18:02:08Z

Its ok, I already applied the patch manually to my Branch. as well as other things.

I discovered this nice tool from GitHub.

https://patch-diff.githubusercontent.com/raw/hgourvest/node-firebird/pull/165.patch

where it shows the patch diff of a particular PR. Sadly I found it after I already copied the code over by hand hahaha.

pcandido · 2020-07-06T18:58:43Z

@hgourvest @mariuz what do you think about this PR? I am facing the same issue and I would like to collaborate. According to the past messages, I understood @r03 has abandoned this PR, but I can assume and solve the conflict. But my question is why it was not merged before (it is open since 2018). Depending on what is the problem I can contribute in order to approve this PR.

leonetosoft · 2020-11-19T18:27:21Z

Perfect tank's!

RonneiPeterson · 2021-07-19T17:52:43Z

I have the same problem, i need put in BD information with acute accent and my field is with charset NONE and collate NONE, but the information save wrong example: café -> CafÃ©

joaodforce · 2021-07-19T18:18:36Z

I have the same problem, i need put in BD information with acute accent and my field is with charset NONE and collate NONE, but the information save wrong example: café -> CafÃ©

you have to be mindful of the charset of the existing data on the DB and the data you are trying to put in.

when the charset is NONE, basically what you put it what you read. The main issue is that this Lib applied UTF8 to everything by default. which would be fine if it was the only thing touching the database.

But if you are mixing access with a Windows Program using WIN-1256 Encoding then we have issues, because the bytes are no encoded in the same way. so if you insert "João" using node, then the windows app will decode UTF-8 as WIN-1256 which will result in "JoÃ©o".

And thats the purpose of this PR, which allows Data to be written as raw Bytes to the Fields, so You can encode the string with WIN-1256 using iconv-lite for example. then the text will be correct everywhere.

My fork of this Repo has this PR applied, altho it is behind several commits, it is still used by my company.

gsimdevelop · 2021-07-21T12:53:28Z

Why this PR was never merged? I have this problem and need it.

RonneiPeterson · 2021-07-21T13:25:10Z

I resolved this problem setting my table fields to charset WIN1252 and in node-firebird i used with out function that convert in buffer and resolved my problem with charset. Em seg., 19 de jul. de 2021 às 14:18, João Eduardo ***@***.***> escreveu:

…

I have the same problem, i need put in BD information with acute accent and my field is with charset NONE and collate NONE, but the information save wrong example: café -> CafÃ© you have to be mindful of the charset of the existing data on the DB and the data you are trying to put in. when the charset is NONE, basically what you put it what you read. The main issue is that this Lib applied UTF8 to everything by default. which would be fine if it was the only thing touching the database. But if you are mixing access with a Windows Program using WIN-1256 Encoding then we have issues, because the bytes are no encoded in the same way. so if you insert "João" using node, then the windows app will decode UTF-8 as WIN-1256 which will result in "JoÃ©o". And thats the purpose of this PR, which allows Data to be written as raw Bytes to the Fields, so You can encode the string with WIN-1256 using iconv-lite for example. then the text will be correct everywhere. My fork of this Repo has this PR applied, altho it is behind several commits, it is still used by my company. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#165 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHPO43YGQ4FTHJ25JAMQVCDTYRUAPANCNFSM4F7YCRAA> .

joaodforce · 2021-07-21T13:28:36Z

I resolved this problem setting my table fields to charset WIN1252 and in node-firebird i used with out function that convert in buffer and resolved my problem with charset. Em seg., 19 de jul. de 2021 às 14:18, João Eduardo @.***> escreveu:
…

Only works if you can make this change, Firebird 2.1 isn't a big fan of Charsets. The library must be agnostic to avoid problems globally.

gsimdevelop · 2021-07-22T06:15:19Z

I can't change the charset of the tables because is a old BD with thousends of procedures, tables, etc. and all the varchars are defined to NONE charset.
Therefore, I haven't another solution.
I tried implement this changes in the actually branch but it not work.

oscarhn27 · 2021-07-23T08:36:28Z

Hi, I also need fix this problem. I can't change de charset of my table's varchars either.

oscarhn27 · 2021-09-01T06:38:01Z

Hi, there are any solutions for this problem or somebody developing in it?
Thank you.

RonneiPeterson · 2021-09-01T08:47:55Z

I converted my fields varchar to charset Win1252 and resolved the problem. Em qua., 1 de set. de 2021 às 02:38, Óscar Hernández Navarro < ***@***.***> escreveu:

…

Hi, there are any solutions for this problem or somebody developing in it? Thank you. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#165 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHPO4352KTVVRYO7N2T5VLLT7XC5JANCNFSM4F7YCRAA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

oscarhn27 · 2021-09-01T09:09:17Z

I know that is a solution but I said that can not change this fields charset because is a very big and old database. Thus this pull request is very helpful.

mreis1 · 2022-08-26T19:28:02Z

Guys, why is this PR still on hold?
Does this PR introduce a breaking change? or something that needs to be handled?

Don't convert param buffer to utf-8

3a56082

When a buffer is used as parameter it would always be converted to a utf-8 string. To allow working with a DB that has charset NONE, the buffer is now send directly without converting to string.

e-baron mentioned this pull request Jan 7, 2019

Insert data with charset NONE (or ISO8859_1) #164

Open

gsimdevelop approved these changes Jul 21, 2021

View reviewed changes

mariuz approved these changes Aug 31, 2022

View reviewed changes

Conversation

ghost commented Oct 28, 2018 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

e-baron commented Jan 7, 2019

Uh oh!

joaodforce commented Mar 31, 2020

Uh oh!

ghost commented Mar 31, 2020

Uh oh!

joaodforce commented Mar 31, 2020

Uh oh!

ghost commented May 2, 2020

Uh oh!

joaodforce commented May 2, 2020

Uh oh!

pcandido commented Jul 6, 2020

Uh oh!

leonetosoft commented Nov 19, 2020

Uh oh!

RonneiPeterson commented Jul 19, 2021

Uh oh!

joaodforce commented Jul 19, 2021

Uh oh!

gsimdevelop commented Jul 21, 2021

Uh oh!

RonneiPeterson commented Jul 21, 2021 via email

Uh oh!

joaodforce commented Jul 21, 2021

Uh oh!

gsimdevelop commented Jul 22, 2021

Uh oh!

oscarhn27 commented Jul 23, 2021

Uh oh!

oscarhn27 commented Sep 1, 2021

Uh oh!

RonneiPeterson commented Sep 1, 2021 via email

Uh oh!

oscarhn27 commented Sep 1, 2021

Uh oh!

mreis1 commented Aug 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

ghost commented Oct 28, 2018 •

edited by ghost

Loading

mreis1 commented Aug 26, 2022 •

edited

Loading